Geographic zone data recovery in geographically distributed data storage environment

ABSTRACT

The described technology is generally directed towards recovery of data segments from geographic zones (dynamic GEO recovery) by having a zone that needs the data direct the recovery process using counterpart segments. If needed data, such as to respond to a client request, is owned by another zone but is lost or corrupt and therefore unavailable from that owning zone, the owning zone instructs the requesting zone to perform recovery. The zone performs recovery by obtaining the counterpart segments, combining (XOR-ing) the counterpart recovery segments into the needed segment, and returns the data to the client. If the zone performing recovery owns one of the counterpart segments, only one of the two counterpart segments needs to be communicated over the inter-zone network, facilitating more efficient, less resource-demanding GEO recovery.

TECHNICAL FIELD

The subject application relates generally to data storage, and, forexample, to a technology that facilitates recovering lost or corruptdata, including in a geographically distributed environment, and relatedembodiments.

BACKGROUND

Contemporary data storage systems, such as Dell EMC®'s ECS (formerlyElastic Cloud Storage) service, store data in a way that ensures dataprotection while retaining storage efficiency. For additional protectionof user data and metadata, ECS supports geographically distributedsetups of multiple zones (geographically distributed node clusters),with the data and metadata of one zone distributed and replicated to twoor more zones by asynchronous replication.

When there are three or more geographic zones, an eXclusive OR (XOR)technique can be used to minimize capacity overhead associated with suchadditional data protection. Instead of storing multiple blocks (such asa chunk) of identically replicated data per zone, one zone can store oneblock of data, another zone can store a different block of data, and yetanother zone can store a third block of data that is a bitwise XOR ofthe two different blocks. For example, consider that some block A ofdata is owned by Zone 1; Zone 1 can store block A, Zone 2 can store a(different) block B, and zone X can store block X, which is block AXOR'ed with block B. Then if block A is ever lost or corrupt, block Acan be restored via an XOR of block X and block B; similarly if bock Bis ever lost or corrupt, block B can be restored via an XOR of block Xand block A.

To be practical in a large data storage system, such data blocks (e.g.,chunks) are relatively large, whereby recovery of a complete data blockinvolving distributed geographic zones can take a relatively long amountof time. Thus, if some amount of data (such as an object) is needed andcannot be returned from a lost or corrupt chunk from its owning zone,rather than wait for the restoration to complete, the owning zonerequests XOR recovery of a segment of data identified by an offset andsize. However, the zone that owns the data is responsible for therecovery of the segment, which can be inefficient in many scenarios.

BRIEF DESCRIPTION OF THE DRAWINGS

The technology described herein is illustrated by way of example and notlimited in the accompanying figures in which like reference numeralsindicate similar elements and in which:

FIG. 1 is an example block diagram representation of part of a datastorage system including nodes and geographic zones, in which geographicrecovery of data can be performed, in accordance with various aspectsand implementations of the subject disclosure

FIGS. 2-4 are example block diagram/data flow diagram representationsrelated to data recovery by a non-owning zone in a distributed zoneenvironment in various scenarios, in accordance with various aspects andimplementations of the subject disclosure.

FIG. 5 is an example block diagram/data flow diagram representationsrelated to data recovery by an owning zone in a distributed zoneenvironment, in accordance with various aspects and implementations ofthe subject disclosure.

FIG. 6 is an example flow diagram showing example operations related tohaving an owning node instruct a requesting zone to implement zone-baseddata recovery on its own, in accordance with various aspects andimplementations of the subject disclosure.

FIG. 7 is an example flow diagram showing example operations of a zonerelated to receiving an instruction to implement zone-based datarecovery, in accordance with various aspects and implementations of thesubject disclosure.

FIG. 8 is an example flow diagram showing example operations of a zonerelated to receiving a client request for data, including when the datais owned but not returnable, in accordance with various aspects andimplementations of the subject disclosure.

FIG. 9 is an example flow diagram showing example operations related toperforming data recovery in a geographic zone when instructed by a zonethat owns the data that the data cannot be returned, in accordance withvarious aspects and implementations of the subject disclosure.

FIG. 10 is an example flow diagram showing example operations related toinstructing a zone that is requesting data to recover the data on itsown, when the data cannot be returned, in accordance with variousaspects and implementations of the subject disclosure.

FIG. 11 is an example flow diagram showing example operations related toperforming data recovery in a geographic zone, by obtaining the recoverydata parts, when instructed by a zone that owns the data that the datacannot be returned, in accordance with various aspects andimplementations of the subject disclosure.

FIG. 12 is a block diagram representing an example computing environmentinto which aspects of the subject matter described herein may beincorporated.

FIG. 13 depicts an example schematic block diagram of a computingenvironment with which the disclosed subject matter can interact/beimplemented at least in part, in accordance with various aspects andimplementations of the subject disclosure.

DETAILED DESCRIPTION

Various aspects of the technology described herein are generallydirected towards reducing inter-zone network traffic when data needs tobe recovered. In one aspect, when a remote zone receives a clientrequest for lost or corrupt data that is owned by another, owning zone,instead of having the owning zone recover and return the data to theremote zone, the owning zone instructs the remote zone to recover andreturn the data to the requesting client. In one scenario, for example,this can reduce the number of times that the data segment/its recoverysegment is communicated across inter-zone boundaries from three to one.

It should be understood that any of the examples herein arenon-limiting. For instance, some of the examples are based on ECS datastorage technology; however virtually any storage system may benefitfrom the technology described herein. As a more particularly example,the term “chunk” can be used as an example of a unit of data storage,however any data block can be used in other storage systems. Similarly,a “segment” identified by an “offset” and “size” is used to indicatepart of a data chunk/block, although it is understood that other termsthat can identify such a sub-unit of storage can be used. Thus, any ofthe embodiments, aspects, concepts, structures, functionalities orexamples described herein are non-limiting, and the technology may beused in various ways that provide benefits and advantages in computingand data storage in general.

Reference throughout this specification to “one embodiment,” “anembodiment,” “one implementation,” “an implementation,” etc. means thata particular feature, structure, or characteristic described inconnection with the embodiment/implementation is included in at leastone embodiment/implementation. Thus, the appearances of such a phrase“in one embodiment,” “in an implementation,” etc. in various placesthroughout this specification are not necessarily all referring to thesame embodiment/implementation. Furthermore, the particular features,structures, or characteristics may be combined in any suitable manner inone or more embodiments/implementations.

Aspects of the subject disclosure will now be described more fullyhereinafter with reference to the accompanying drawings in which examplecomponents, graphs and/or operations are shown. In the followingdescription, for purposes of explanation, numerous specific details areset forth in order to provide a thorough understanding of the variousembodiments. However, the subject disclosure may be embodied in manydifferent forms and should not be construed as limited to the examplesset forth herein.

In ECS, disk space is partitioned into a set of blocks of fixed sizecalled chunks, which in one or more implementations are 128 megabytes insize. The various types of data, including user data and various typesof metadata, are stored in chunks. There are different types of chunks,one type per capacity user. In particular, user data is stored inrepository chunks, and chunks can be shared. For instance, one chunk may(and in typical cases does) contain segments of multiple user objects.

As set forth herein, geographic zones can be used to replicate data,including user chunks, for additional data protection. The various userdata chunks are distributed among the zones, with one zone (a node inthe zone cluster) responsible for owning a given chunk. Becausereplication takes time (and because in environments having three or morezones the data is not directly available at another zone), a clientrequest to one zone for data that is in a chunk owned by another zone isobtained by having the other zone request and receive the data from theowning zone, and then once received return the data to the requestingclient.

However, when the requested data is not available from the zone thatowns the chunk, the requested data needs to be recovered. As set forthherein, recovery of a complete chunk (e.g., 128 MB) can take time, sothe needed segment is recovered separately. For data storageenvironments having three or more zones, XOR can be used; for Zone 1(which owns Chunk A) and Zone 2 (which owns Chunk B), both zones canreplicate their respective chunks A and B to Zone 3. Zone 3 does notstore chunk copies for Chunk A and Chunk B but instead only one Chunk Xis stored by Zone 3, comprising the result of XOR (eXclusive OR) forChunk A content and Chunk B content, that is, Chunk X=XOR(Chunk A, ChunkB).

When a chunk with user data, e.g., Chunk A or Chunk B, is unavailable,the corresponding XOR chunk can be used to restore its content via GEOrecovery. GEO recovery can be represented as:

Chunk A=XOR(Chunk X,Chunk B), and

Chunk B=XOR(Chunk X,Chunk A).

In such a setup, Chunk A contains an object segment, with the segment'scontent represented as Chunk A(offset, size). Then, if Chunk A islost/corrupt, the object segment can be quickly recovered using(relatively small) parts of Chunk X and Chunk B, by:

Chunk A(offset,size)=XOR(Chunk X(offset,size),Chunk B(offset,size)).

Given the above examples, consider that a client requests an object fromZone 2 in which the objects corresponds to the Chunk A(offset, size)segment owned by Zone 1. Zone 2, which contains information thatindicates that Zone 1 owns chunk A, requests the data segment(abbreviated to A(o,s)) from Zone 1. In this example, consider thatChunk A is lost or corrupt; when this occurs, Zone 1 requests Zone 3 torecover A(o,s) using its XOR-ed Chunk X. Zone 3 recognizes that B(o,s))is needed to do this, and thus requests and receives B(o,s)) from zone2; (note that this is a first time the segment-related data of size s,of counterpart segment B(o,s) is communicated from one zone to another).

Zone 3 then uses its Chunk X to XOR X(o,s) with B(o,s), and therebyreturn recovered chunk A(0,s) to Zone 1; (note that this is a secondtime the segment-related data of size s, of recovered segment A(o,s) iscommunicated from one zone to another). Zone 1 then returns recoveredsegment A(o,s) to Zone 2, (note that this is a third time thesegment-related data of size s, of recovered segment A(o,s) iscommunicated from one zone to another). Zone 2 in turn responds to theclient with A(o,s) to complete the client request.

As is understood, there are various cycles of data requests and datasegments transmissions. In particular, data segments of size s aretransmitted over inter-zone network three times in the above scenario.Described herein is a technology that facilitates more efficientgeographic data recovery, by having the zone that receives a data readrequest from a client perform the data recovery, which as will beunderstood, reduces the number of times that the data segments of size sneed to be transmitted across zones.

FIG. 1 shows part of a data storage system 100 (such as ECS) comprisinga node cluster 102 of storage nodes 104(1)-104(M), in which each node istypically a server configured primarily to serve objects in response toclient requests. The nodes 104(1)-104(M) are coupled to each other via asuitable data communications link comprising interfaces and protocols,such as represented in FIG. 1 by Ethernet block 106.

Clients 108 make data system-related requests to the cluster 102, whichin general is configured as one large object namespace; there may be onthe order of billions of objects maintained in a cluster, for example.To this end, a node such as the node 104(2) (shown enlarged in FIG. 1 aswell) generally comprises ports 112 by which clients connect to thecloud storage system. Example ports are provided for requests viavarious protocols, including but not limited to SMB (server messageblock), FTP (file transfer protocol), HTTP/HTTPS (hypertext transferprotocol) and NFS (Network File System); further, SSH (secure shell)allows administration-related requests, for example.

Each node, such as the node 104(2), includes an instance of a datastorage system and data services 114; (note however that at least somedata service components can be per-cluster, rather than per-node). Forexample, ECS runs a set of storage services, which together implementstorage logic. Services can maintain directory tables for keeping theirmetadata, which can be implemented as search trees. A blob service 116maintains an object table 118 (e.g., in various partitions among nodes,including geographically separated zones) that keeps track of objects inthe data storage system and generally stores their metadata, includingan object's data location information, e.g., within a chunk. The blobservice 116 also maintains a listing table 120, although it isalternatively feasible to have such a listing table maintained byanother service.

FIG. 1 further represents some additional concepts, in that the userdata repository of chunks is maintained in a chunk store 122, managed byanother storage service referred to as a chunk manager 124. A chunktable 126 maintains metadata about chunks, e.g., as managed by the chunkmanager 124. Note that directory tables and other data can also bemaintained in data chunks.

In one or more implementations, the data services 114 can also includegeographic-related services (block 128), such as replication and (asdescribed herein) geo-recovery related communications to and from remotezones 130 and their data storage 131. As is understood, sending databetween a local zone and a remote zone is relatively inefficient, andare thus reduced to the extent possible via the technology describedherein.

In FIG. 1, a CPU 132 and RAM 134 are shown for completeness; note thatthe RAM 132 may comprise at least some non-volatile RAM. The node 104(2)further includes storage devices such as disks 136, comprising hard diskdrives and/or solid-state drives, or any other suitable type of storageresource. As can be readily appreciated, components of the data storagesystem including those described herein can be at various times in anystorage device or devices, such as in the RAM 134, in the disks 136, orin a combination of both, for example.

As represented in FIG. 2, in an example implementation similar to theabove example(s), a read request for an object's data is received from aclient 208 (the arrow labeled one (1)) at a Zone B 222. As before, inthis example Zone 1 221 owns Chunk A 227, Zone 2 222 owns Chunk B 228,and Zone 3 223 own chunk X 229 based on the XOR-ing of replicated copiesof respective chunk A 227 and chunk B 228 to Zone 3 223.

When the read request is received and processed, Zone 2 222 requests therelevant segment and chunk associated with the requested object, and atarrow two (2) sends a request for the segment “get A(0,s)” to the chunkA's owner, Zone 1 222. The Zone A 221 detects that the Chunk A's datacannot be returned (Chunk A 227 is lost/corrupt, as indicated by thelarge crossed lines in the upper right corner of the block representingchunk A 229), and responds with an indication that the data cannot bereturned, such as a communication (arrow three (3)) instructing Zone B222 to get the data itself. Such a “get-data-yourself” instruction canbe an errorcode or the like understood by the various zones.

Chunk B receives the “get data yourself” instruction, accesses itsinformation indicating that Zone 3 223 contains the needed XOR recoverypart of the segment in chunk X, and requests X(o,s) from the Zone 3 223,as represented by the labeled arrow four (4). Zone 3 responds with thecounterpart recovery segment X(o,s) at arrow five (5). Note that in FIG.2, this is the first (and only) time that data of size s is communicatedacross zones.

Note that depending on storage system implementation specifics, the“get-data-yourself” instruction can be accompanied by furtherinstructions on how the recovery can be performed. For example, the Zone1 can instruct the requesting zone as to which other zones contain therecovery parts, in this example Zone 2 and Zone 3.

Once Zone 2 222 receives segment X(o,s) from the zone 3 223, therequested segment data is recovered by XOR-ing segment X(o,s) with ChunkB's counterpart segment B(o,s). The requested object is thus returned tothe client 208 as represented in FIG. 2 by the labeled arrow six (6).

As can be seen, the initial scenario for segment A(o,s) is the same aspreviously handled, but the zones act differently. Instead of Zone 1driving data recovery when Zone 1 detects corruption/loss of its ChunkA, Zone 1 instructs Zone 2 to drive the recovery. From this instruction,Zone 2 realizes that the data is unreturnable from Zone 1, whereby Zone2 takes charge of on-the-fly GEO recovery of the data. Note that Zone 2already has the segment B(o,s) in Chunk B 228, and that segment X(o,s)is needed for the recovery. Zone 2 reads X(o,s) from Zone 3, XOR-s thissegment with local segment B(o,s), and sends the result, which is A(o,s)and the object data at the same time, to the data client. Note that anyof the segment-related information can be cached for some appropriatetime, so that for example if the Zone 3 receives another requestcorresponding to segment A(o,s), no similar GEO recovery is needed;(note that such caching is feasible in other scenarios, such as if Zone1 was able to return the segment A(o,s)).

To summarize, the GEO recovery path that is driven by Zone 2 is muchshorter than the one described above in which the owning zone drove therecovery. In particular, in the example implementation of FIG. 2, onlyone data segment of size s is transmitted over the inter-zone network,instead of three data segments.

Note that Zone 1 221 may also initiate full GEO recovery of completeChunk A in conjunction with replying with the “get-data-yourself”instruction to Zone 2. Such full GEO recovery of Chunk A typicallyfinishes long after the data read request is served.

The example shown in FIG. 3 is similar to FIG. 2, except that in FIG. 3,Zone 3 223 receives the client request for the object. Thus, when Zone 3223 receives the “get data yourself” instruction from chunk A, Zone 3obtains segment B(o,s) from Zone 2, performs the XOR with its ownsegment part from Chunk X 229 to obtain A(o,s) and returns thecorresponding object data to the requesting client.

FIG. 4 shows a four zone scenario, in which a zone 4 224 that owns norecovery part of the segment receives the request for the object fromsome client 408. As can be seen by following the labeled arrows, byhaving Zone 4 224 drive the recovery via the “get-data-yourself”instruction at arrow (3), only two segments of size s need to becommunicated across zones, instead of three such segments. That is,inter-zone traffic is reduced because there is no need for Zone 1 221 toget the recovered data segment A(o,s) to Zone 4 224, because Zone D 224gets the segment parts needed to perform the XOR recovery.

FIG. 5 shows a modified scenario in which the Zone that owns the chunkfor a requested segment receives the request for the object, e.g., Zone1 221 receives an object request for data corresponding to segmentA(o,s), such as from another client 508. In this example, Zone A drivesthe recovery, but instead of having Zone 3 obtain chunk segment B(o,s),chunk A requests B(o,s) from Zone 2 222 (arrows 2 and 4) and X(o,s) fromZone 3 223 (arrow 3 and 5). This does not particularly reduce inter-zonecommunications, but allows for generally parallel requesting of theneeded recovery data parts, which can be more efficient.

Whether to have Zone X be responsible for the recovery (and returnrecovered segment A(o,s) as in prior solutions) or to have Zone A beresponsible for GEO recovery as in FIG. 5 can be dependent on otherfactors. Consider for example that Zone 1 is geographically between Zone2 and Zone 3; it can be faster for Zone 1 to receive segment B(o,s) fromZone 2 while receiving segment X(o,s) from Zone 3 instead of waiting forZone 3 to obtain B(o,s) from Zone 2. If instead Zone 3 is geographicallybetween Zone 1 and Zone 3; it may be faster for Zone 3 to receivesegment B(o,s) from Zone 2. Other factors, such as the speeds ofcommunications links, can be considered. Still other factors such asrelative zone workloads can be considered, e.g., a zone that is heavilybusy such as during a busy workday can offload recovery to one of theother zones, such as one where it is nighttime and handling far lesswork.

FIG. 6 shows example operations related to the “get data yourself”instruction, beginning at operation 602 where a request for owned datais received from a remote zone. If the data is returnable as evaluatedat operation 604, then the requested data is returned at operation 606.

In FIG. 6, if at operation 604 the data is not returnable (as in FIGS.2-5), e.g., is lost or corrupt, operation 608 instructs the requestingremote zone to perform the GEO data recovery operation as describedherein. FIGS. 7 and 8 are directed towards operations of the other zonethat receives the instruction.

Operation 610 represents the owning node initiating complete recovery ofthe chunk. Note that operation 610 is optional at this time, as it canbe performed in a separate process/set of operations as in existingsystems. It is also feasible that full recovery was previously initiated(but not yet completed) as a result of a prior request for some segmentdata in that chunk, e.g., A(o's′) (which could be the same segmentA(o,s)).

FIG. 7 shows example operations of a zone, which in response to arequest for data from an owning zone, receives the instruction torecover and return the data on its own. Operation 704 representsdetermining the zone that has the second part of the recovery data, thatis, the XOR part from its perspective, which the zone requests atoperation 706.

Operation 708 evaluates whether the zone that is driving recovery hasthe first part of the recovery data (as in FIG. 2), or another zone hasthis part (as in FIG. 4). If the zone owns the first part, operation 710obtains this data, and branches ahead to operation 718 to await thereceiving of the second part.

Otherwise, operation 714 is performed to request the first part, whichis received (after some delay) at operation 716. When both parts arereceived, whether because of ownership at operation 710 or via therequest at operation 714, operation 720 recovers (XOR-s) the data parts.Operation 722 represents returning the recovered data to the requestingentity, e.g., the client, or possibly another zone.

FIG. 8 shows operations that occur when a zone receives a request from aclient to return an object. If the requested data is not owned asevaluated by operation 804, then the remote zone that owns the neededdata is determined (operation 808) and a request for the data is madefrom the remote zone (operation 810). If the data is received asevaluated at operation 812, the data is returned to the client atoperation 814, and the process ends. If instead the (“get datayourself”) instruction to recover the data is received, the operationsof FIG. 7 can be performed, as described herein.

Returning to operation 804, if the requested data is owned, operation806 evaluates whether the data is returnable, and if so, the data isreturned to the client via operation 814. If the data is owned but notreturnable (as in the example of FIG. 5), operations 816 and 818determine the zones that own the recovery data parts, and request theparts from those zones.

When the first and second parts of the data are received as representedby operation 820, operation 822 recovers (XOR-s) the data parts, andoperation 824 returns the data to the client. Operation 826 optionallyinitiates full recovery of the data chunk (if not already initiated, forexample).

As set forth above, instead of taking the no branch of operation 806 (asin the example of FIG. 5), it is feasible to have another zone, such asthe zone that owns the XOR chunk, drive the recovery operation (e.g.,instruct that zone to recover and return the segment) according toexisting solutions. To reiterate, offloading segment recovery can bedependent on other factors, such as relative locations of the zones,speed of communication links between zones, relative zone workloads, andso on.

One or more aspects can be embodied in a system, such as represented inFIG. 9, and for example can comprise a memory that stores computerexecutable components and/or operations, and a processor that executescomputer executable components and/or operations stored in the memory.Example operations can comprise operation 902, which representsreceiving, at a local distributed zone of a data storage system ofdistributed zones, a client request for requested data from a client.Operation 904 represents, based on determining that the requested datais owned by a first remote distributed zone, requesting the requesteddata from the first remote distributed zone. Operation 906 representsreceiving an indication from the first remote distributed zone that therequested data is not returnable from the first remote distributed zone.Operation 908 represents, in response to the receiving the indication,operation 910, which represents obtaining first recovery data, operation910, which represents obtaining second recovery data and operation 912,which represents combining the first recovery data and the secondrecovery data to obtain the requested data. Operation 916 representssending, in response to the client request, the requested data from thelocal distributed zone to the client.

Obtaining the first recovery data can comprise accessing a storagedevice of the local distributed zone; obtaining the second recovery datacan comprise requesting the second recovery data from a second remotedistributed zone and receiving the second recovery data from a secondremote distributed zone.

Further operations can comprise receiving a recovery request from thefirst remote distributed zone to recover a copy of a lost or corruptdata structure owned by the first remote distributed zone that storesthe requested data.

Obtaining the second recovery data can comprise requesting the secondrecovery data from a second remote distributed zone and receiving thesecond recovery data from the second remote distributed zone; obtainingthe first recovery data can comprise requesting the first recovery datafrom a third remote distributed zone and receiving the first recoverydata from the third remote distributed zone.

Combining the first recovery data and the second recovery data cancomprise performing a bitwise logical XOR operation of the firstrecovery data and the second recovery data to obtain the requested data.The client request for the requested data can correspond to a datasegment in a chunk that stores the requested data.

Requesting the requested data from the first remote distributed zone cancomprise identifying the chunk, and an offset value and a size valuerepresenting the data segment.

The chunk can be a first chunk, the first recovery data can correspondto a first counterpart data segment in a second chunk, and the secondrecovery data can correspond to a second counterpart data segment in athird chunk in which the third chunk comprises a bitwise XOR combinationof the first chunk and the second chunk. One or more example aspects,such as corresponding to operations of a method, are represented in FIG.10. Operation 1002 represents receiving, by a system comprising aprocessor in a first geographically distributed zone, a request from asecond geographically distributed zone for requested data owned by thefirst geographically distributed zone. Operation 1004 representsdetermining, by the system in the first geographically distributed zone,that the requested data is not returnable from the first geographicallydistributed zone. Operation 1006 represents, in response to thedetermining, instructing, by the system in the first geographicallydistributed zone, the second geographic zone to recover the requesteddata.

The request can be a first request, the requested data can be firstrequested data, and aspects can comprise receiving, by the system in thefirst geographically distributed zone, a second request from a clientrequester for second requested data owned by the first geographicallydistributed zone, determining, by the system in the first geographicallydistributed zone, that the second requested data is not returnable fromthe first geographically distributed zone, and in response to thedetermining, instructing, by the system in the first geographicallydistributed zone, the second geographic zone to provide a first recoverypart of the requested data, instructing, by the system in the firstgeographically distributed zone, a third geographic zone to provide asecond recovery part of the requested data, receiving, by the system inthe first geographically distributed zone, the first recovery part,receiving, by the system in the first geographically distributed zone,the second recovery part, recovering, by the system in the firstgeographically distributed zone, the second requested data by combiningthe first recovery part and the second recovery part, and returning, bythe system in the first geographically distributed zone, the secondrequested data to the client requester in response to the secondrequest.

Recovering the second requested data by combining the first recoverypart and the second recovery part can comprise performing an XORoperation. The requested data can be part of a corrupt data storagechunk owned by the first geographically distributed zone, and aspectscan comprise initiating, by the system in the first geographicallydistributed zone, recovery of a non-corrupt replacement copy of thecorrupt data storage chunk. The requested data can be part of a lostdata storage chunk owned by the first geographically distributed zone,and aspects can comprise, initiating, by the system in the firstgeographically distributed zone, recovery of a replacement copy of thelost data storage chunk.

FIG. 11 summarizes various example operations, e.g., corresponding to amachine-readable storage medium, comprising executable instructionsthat, when executed by a processor of a system in a second distributedzone of a data storage system of geographic zones, facilitateperformance of operations. Operation 1102 represents receiving a clientrequest for requested data owned by a first distributed zone. Operation1104 represents, in response to the client request, requesting therequested data from the first distributed zone. Operation 1106represents receiving an indication from the first distributed zone thatthe requested data is not returnable from the first distributed zone.Operation 1108 represents obtaining first recovery data. Operation 1110represents obtaining second recovery data from a third distributed zone.Operation 1112 represents combining the first recovery data and thesecond recovery data to obtain the requested data. Operation 1114represents returning the requested data from the second distributed zonein response to the client request

Obtaining the first recovery data can comprise accessing a storagedevice of the second distributed zone. Obtaining the first recovery datacan comprise requesting and receiving the first recovery data from afourth distributed zone.

Combining the first recovery data and the second recovery data cancomprise performing a bitwise XOR operation of the first recovery dataand the second recovery data to obtain the requested data.

Receiving the client request can comprise receiving a request for anobject that corresponds to a data segment in a data chunk owned by thefirst distributed zone.

The chunk can be a first chunk, obtaining the first recovery data cancomprise accessing a first counterpart data segment maintained in asecond chunk owned by the second distributed zone, and obtaining thesecond recovery data from the third distributed zone can compriserequesting a second counterpart data segment maintained in a third chunkowned by the third distributed zone.

The chunk can be a first chunk, obtaining the first recovery data cancomprise requesting a first counterpart data segment maintained in asecond chunk owned by a fourth distributed zone, and obtaining thesecond recovery data from the third distributed zone can compriserequesting a second counterpart data segment maintained in a third chunkowned by the third distributed zone.

As can be seen, described herein is technology for more efficient GEOrecovery by using less resources/less inter-zone data traffic. By havinga zone that needs data owned by, but unavailable from a remote zone,perform the GEO recovery, the amount of data segments needed to betransmitted over inter-zone network for recovery can be reduced, e.g.,from three to one if the requesting zone owns one of the counterpartrecovery segments, or from three to two if the requesting zone needs torequest both counterpart recovery segments.

FIG. 12 is a schematic block diagram of a computing environment 1200with which the disclosed subject matter can interact. The system 1200comprises one or more remote component(s) 1210. The remote component(s)1210 can be hardware and/or software (e.g., threads, processes,computing devices). In some embodiments, remote component(s) 1210 can bea distributed computer system, connected to a local automatic scalingcomponent and/or programs that use the resources of a distributedcomputer system, via communication framework 1240. Communicationframework 1240 can comprise wired network devices, wireless networkdevices, mobile devices, wearable devices, radio access network devices,gateway devices, femtocell devices, servers, etc.

The system 1200 also comprises one or more local component(s) 1220. Thelocal component(s) 1220 can be hardware and/or software (e.g., threads,processes, computing devices). In some embodiments, local component(s)1220 can comprise an automatic scaling component and/or programs thatcommunicate/use the remote resources 1210 and 1220, etc., connected to aremotely located distributed computing system via communicationframework 1240.

One possible communication between a remote component(s) 1210 and alocal component(s) 1220 can be in the form of a data packet adapted tobe transmitted between two or more computer processes. Another possiblecommunication between a remote component(s) 1210 and a localcomponent(s) 1220 can be in the form of circuit-switched data adapted tobe transmitted between two or more computer processes in radio timeslots. The system 1200 comprises a communication framework 1240 that canbe employed to facilitate communications between the remote component(s)1210 and the local component(s) 1220, and can comprise an air interface,e.g., Uu interface of a UMTS network, via a long-term evolution (LTE)network, etc. Remote component(s) 1210 can be operably connected to oneor more remote data store(s) 1250, such as a hard drive, solid statedrive, SIM card, device memory, etc., that can be employed to storeinformation on the remote component(s) 1210 side of communicationframework 1240. Similarly, local component(s) 1220 can be operablyconnected to one or more local data store(s) 1230, that can be employedto store information on the local component(s) 1220 side ofcommunication framework 1240.

In order to provide additional context for various embodiments describedherein, FIG. 13 and the following discussion are intended to provide abrief, general description of a suitable computing environment 1300 inwhich the various embodiments of the embodiment described herein can beimplemented. While the embodiments have been described above in thegeneral context of computer-executable instructions that can run on oneor more computers, those skilled in the art will recognize that theembodiments can be also implemented in combination with other programmodules and/or as a combination of hardware and software.

Generally, program modules include routines, programs, components, datastructures, etc., that perform particular tasks or implement particularabstract data types. Moreover, those skilled in the art will appreciatethat the inventive methods can be practiced with other computer systemconfigurations, including single-processor or multiprocessor computersystems, minicomputers, mainframe computers, Internet of Things (IoT)devices, distributed computing systems, as well as personal computers,hand-held computing devices, microprocessor-based or programmableconsumer electronics, and the like, each of which can be operativelycoupled to one or more associated devices.

The illustrated embodiments of the embodiments herein can be alsopracticed in distributed computing environments where certain tasks areperformed by remote processing devices that are linked through acommunications network. In a distributed computing environment, programmodules can be located in both local and remote memory storage devices.

Computing devices typically include a variety of media, which caninclude computer-readable storage media, machine-readable storage media,and/or communications media, which two terms are used herein differentlyfrom one another as follows. Computer-readable storage media ormachine-readable storage media can be any available storage media thatcan be accessed by the computer and includes both volatile andnonvolatile media, removable and non-removable media. By way of example,and not limitation, computer-readable storage media or machine-readablestorage media can be implemented in connection with any method ortechnology for storage of information such as computer-readable ormachine-readable instructions, program modules, structured data orunstructured data.

Computer-readable storage media can include, but are not limited to,random access memory (RAM), read only memory (ROM), electricallyerasable programmable read only memory (EEPROM), flash memory or othermemory technology, compact disk read only memory (CD-ROM), digitalversatile disk (DVD), Blu-ray disc (BD) or other optical disk storage,magnetic cassettes, magnetic tape, magnetic disk storage or othermagnetic storage devices, solid state drives or other solid statestorage devices, or other tangible and/or non-transitory media which canbe used to store desired information. In this regard, the terms“tangible” or “non-transitory” herein as applied to storage, memory orcomputer-readable media, are to be understood to exclude onlypropagating transitory signals per se as modifiers and do not relinquishrights to all standard storage, memory or computer-readable media thatare not only propagating transitory signals per se.

Computer-readable storage media can be accessed by one or more local orremote computing devices, e.g., via access requests, queries or otherdata retrieval protocols, for a variety of operations with respect tothe information stored by the medium.

Communications media typically embody computer-readable instructions,data structures, program modules or other structured or unstructureddata in a data signal such as a modulated data signal, e.g., a carrierwave or other transport mechanism, and includes any information deliveryor transport media. The term “modulated data signal” or signals refersto a signal that has one or more of its characteristics set or changedin such a manner as to encode information in one or more signals. By wayof example, and not limitation, communication media include wired media,such as a wired network or direct-wired connection, and wireless mediasuch as acoustic, RF, infrared and other wireless media.

With reference again to FIG. 13, the example environment 1300 forimplementing various embodiments of the aspects described hereinincludes a computer 1302, the computer 1302 including a processing unit1304, a system memory 1306 and a system bus 1308. The system bus 1308couples system components including, but not limited to, the systemmemory 1306 to the processing unit 1304. The processing unit 1304 can beany of various commercially available processors. Dual microprocessorsand other multi-processor architectures can also be employed as theprocessing unit 1304.

The system bus 1308 can be any of several types of bus structure thatcan further interconnect to a memory bus (with or without a memorycontroller), a peripheral bus, and a local bus using any of a variety ofcommercially available bus architectures. The system memory 1306includes ROM 1310 and RAM 1312. A basic input/output system (BIOS) canbe stored in a non-volatile memory such as ROM, erasable programmableread only memory (EPROM), EEPROM, which BIOS contains the basic routinesthat help to transfer information between elements within the computer1302, such as during startup. The RAM 1312 can also include a high-speedRAM such as static RAM for caching data.

The computer 1302 further includes an internal hard disk drive (HDD)1314 (e.g., EIDE, SATA), and can include one or more external storagedevices 1316 (e.g., a magnetic floppy disk drive (FDD) 1316, a memorystick or flash drive reader, a memory card reader, etc.). While theinternal HDD 1314 is illustrated as located within the computer 1302,the internal HDD 1314 can also be configured for external use in asuitable chassis (not shown). Additionally, while not shown inenvironment 1300, a solid state drive (SSD) could be used in additionto, or in place of, an HDD 1314.

Other internal or external storage can include at least one otherstorage device 1320 with storage media 1322 (e.g., a solid state storagedevice, a nonvolatile memory device, and/or an optical disk drive thatcan read or write from removable media such as a CD-ROM disc, a DVD, aBD, etc.). The external storage 1316 can be facilitated by a networkvirtual machine. The HDD 1314, external storage device(s) 1316 andstorage device (e.g., drive) 1320 can be connected to the system bus1308 by an HDD interface 1324, an external storage interface 1326 and adrive interface 1328, respectively.

The drives and their associated computer-readable storage media providenonvolatile storage of data, data structures, computer-executableinstructions, and so forth. For the computer 1302, the drives andstorage media accommodate the storage of any data in a suitable digitalformat. Although the description of computer-readable storage mediaabove refers to respective types of storage devices, it should beappreciated by those skilled in the art that other types of storagemedia which are readable by a computer, whether presently existing ordeveloped in the future, could also be used in the example operatingenvironment, and further, that any such storage media can containcomputer-executable instructions for performing the methods describedherein.

A number of program modules can be stored in the drives and RAM 1312,including an operating system 1330, one or more application programs1332, other program modules 1334 and program data 1336. All or portionsof the operating system, applications, modules, and/or data can also becached in the RAM 1312. The systems and methods described herein can beimplemented utilizing various commercially available operating systemsor combinations of operating systems.

Computer 1302 can optionally comprise emulation technologies. Forexample, a hypervisor (not shown) or other intermediary can emulate ahardware environment for operating system 1330, and the emulatedhardware can optionally be different from the hardware illustrated inFIG. 13. In such an embodiment, operating system 1330 can comprise onevirtual machine (VM) of multiple VMs hosted at computer 1302.Furthermore, operating system 1330 can provide runtime environments,such as the Java runtime environment or the .NET framework, forapplications 1332. Runtime environments are consistent executionenvironments that allow applications 1332 to run on any operating systemthat includes the runtime environment. Similarly, operating system 1330can support containers, and applications 1332 can be in the form ofcontainers, which are lightweight, standalone, executable packages ofsoftware that include, e.g., code, runtime, system tools, systemlibraries and settings for an application.

Further, computer 1302 can be enable with a security module, such as atrusted processing module (TPM). For instance with a TPM, bootcomponents hash next in time boot components, and wait for a match ofresults to secured values, before loading a next boot component. Thisprocess can take place at any layer in the code execution stack ofcomputer 1302, e.g., applied at the application execution level or atthe operating system (OS) kernel level, thereby enabling security at anylevel of code execution.

A user can enter commands and information into the computer 1302 throughone or more wired/wireless input devices, e.g., a keyboard 1338, a touchscreen 1340, and a pointing device, such as a mouse 1342. Other inputdevices (not shown) can include a microphone, an infrared (IR) remotecontrol, a radio frequency (RF) remote control, or other remote control,a joystick, a virtual reality controller and/or virtual reality headset,a game pad, a stylus pen, an image input device, e.g., camera(s), agesture sensor input device, a vision movement sensor input device, anemotion or facial detection device, a biometric input device, e.g.,fingerprint or iris scanner, or the like. These and other input devicesare often connected to the processing unit 1304 through an input deviceinterface 1344 that can be coupled to the system bus 1308, but can beconnected by other interfaces, such as a parallel port, an IEEE 1394serial port, a game port, a USB port, an IR interface, a BLUETOOTH®interface, etc.

A monitor 1346 or other type of display device can be also connected tothe system bus 1308 via an interface, such as a video adapter 1348. Inaddition to the monitor 1346, a computer typically includes otherperipheral output devices (not shown), such as speakers, printers, etc.

The computer 1302 can operate in a networked environment using logicalconnections via wired and/or wireless communications to one or moreremote computers, such as a remote computer(s) 1350. The remotecomputer(s) 1350 can be a workstation, a server computer, a router, apersonal computer, portable computer, microprocessor-based entertainmentappliance, a peer device or other common network node, and typicallyincludes many or all of the elements described relative to the computer1302, although, for purposes of brevity, only a memory/storage device1352 is illustrated. The logical connections depicted includewired/wireless connectivity to a local area network (LAN) 1354 and/orlarger networks, e.g., a wide area network (WAN) 1356. Such LAN and WANnetworking environments are commonplace in offices and companies, andfacilitate enterprise-wide computer networks, such as intranets, all ofwhich can connect to a global communications network, e.g., theInternet.

When used in a LAN networking environment, the computer 1302 can beconnected to the local network 1354 through a wired and/or wirelesscommunication network interface or adapter 1358. The adapter 1358 canfacilitate wired or wireless communication to the LAN 1354, which canalso include a wireless access point (AP) disposed thereon forcommunicating with the adapter 1358 in a wireless mode.

When used in a WAN networking environment, the computer 1302 can includea modem 1360 or can be connected to a communications server on the WAN1356 via other means for establishing communications over the WAN 1356,such as by way of the Internet. The modem 1360, which can be internal orexternal and a wired or wireless device, can be connected to the systembus 1308 via the input device interface 1344. In a networkedenvironment, program modules depicted relative to the computer 1302 orportions thereof, can be stored in the remote memory/storage device1352. It will be appreciated that the network connections shown areexample and other means of establishing a communications link betweenthe computers can be used.

When used in either a LAN or WAN networking environment, the computer1302 can access cloud storage systems or other network-based storagesystems in addition to, or in place of, external storage devices 1316 asdescribed above. Generally, a connection between the computer 1302 and acloud storage system can be established over a LAN 1354 or WAN 1356e.g., by the adapter 1358 or modem 1360, respectively. Upon connectingthe computer 1302 to an associated cloud storage system, the externalstorage interface 1326 can, with the aid of the adapter 1358 and/ormodem 1360, manage storage provided by the cloud storage system as itwould other types of external storage. For instance, the externalstorage interface 1326 can be configured to provide access to cloudstorage sources as if those sources were physically connected to thecomputer 1302.

The computer 1302 can be operable to communicate with any wirelessdevices or entities operatively disposed in wireless communication,e.g., a printer, scanner, desktop and/or portable computer, portabledata assistant, communications satellite, any piece of equipment orlocation associated with a wirelessly detectable tag (e.g., a kiosk,news stand, store shelf, etc.), and telephone. This can include WirelessFidelity (Wi-Fi) and BLUETOOTH® wireless technologies. Thus, thecommunication can be a predefined structure as with a conventionalnetwork or simply an ad hoc communication between at least two devices.

The above description of illustrated embodiments of the subjectdisclosure, comprising what is described in the Abstract, is notintended to be exhaustive or to limit the disclosed embodiments to theprecise forms disclosed. While specific embodiments and examples aredescribed herein for illustrative purposes, various modifications arepossible that are considered within the scope of such embodiments andexamples, as those skilled in the relevant art can recognize.

In this regard, while the disclosed subject matter has been described inconnection with various embodiments and corresponding Figures, whereapplicable, it is to be understood that other similar embodiments can beused or modifications and additions can be made to the describedembodiments for performing the same, similar, alternative, or substitutefunction of the disclosed subject matter without deviating therefrom.Therefore, the disclosed subject matter should not be limited to anysingle embodiment described herein, but rather should be construed inbreadth and scope in accordance with the appended claims below.

As it employed in the subject specification, the term “processor” canrefer to substantially any computing processing unit or devicecomprising, but not limited to comprising, single-core processors;single-processors with software multithread execution capability;multi-core processors; multi-core processors with software multithreadexecution capability; multi-core processors with hardware multithreadtechnology; parallel platforms; and parallel platforms with distributedshared memory. Additionally, a processor can refer to an integratedcircuit, an application specific integrated circuit, a digital signalprocessor, a field programmable gate array, a programmable logiccontroller, a complex programmable logic device, a discrete gate ortransistor logic, discrete hardware components, or any combinationthereof designed to perform the functions described herein. Processorscan exploit nano-scale architectures such as, but not limited to,molecular and quantum-dot based transistors, switches and gates, inorder to optimize space usage or enhance performance of user equipment.A processor may also be implemented as a combination of computingprocessing units.

As used in this application, the terms “component,” “system,”“platform,” “layer,” “selector,” “interface,” and the like are intendedto refer to a computer-related entity or an entity related to anoperational apparatus with one or more specific functionalities, whereinthe entity can be either hardware, a combination of hardware andsoftware, software, or software in execution. As an example, a componentmay be, but is not limited to being, a process running on a processor, aprocessor, an object, an executable, a thread of execution, a program,and/or a computer. By way of illustration and not limitation, both anapplication running on a server and the server can be a component. Oneor more components may reside within a process and/or thread ofexecution and a component may be localized on one computer and/ordistributed between two or more computers. In addition, these componentscan execute from various computer readable media having various datastructures stored thereon. The components may communicate via localand/or remote processes such as in accordance with a signal having oneor more data packets (e.g., data from one component interacting withanother component in a local system, distributed system, and/or across anetwork such as the Internet with other systems via the signal). Asanother example, a component can be an apparatus with specificfunctionality provided by mechanical parts operated by electric orelectronic circuitry, which is operated by a software or a firmwareapplication executed by a processor, wherein the processor can beinternal or external to the apparatus and executes at least a part ofthe software or firmware application. As yet another example, acomponent can be an apparatus that provides specific functionalitythrough electronic components without mechanical parts, the electroniccomponents can comprise a processor therein to execute software orfirmware that confers at least in part the functionality of theelectronic components.

In addition, the term “or” is intended to mean an inclusive “or” ratherthan an exclusive “or.” That is, unless specified otherwise, or clearfrom context, “X employs A or B” is intended to mean any of the naturalinclusive permutations. That is, if X employs A; X employs B; or Xemploys both A and B, then “X employs A or B” is satisfied under any ofthe foregoing instances.

While the invention is susceptible to various modifications andalternative constructions, certain illustrated implementations thereofare shown in the drawings and have been described above in detail. Itshould be understood, however, that there is no intention to limit theinvention to the specific forms disclosed, but on the contrary, theintention is to cover all modifications, alternative constructions, andequivalents falling within the spirit and scope of the invention.

In addition to the various implementations described herein, it is to beunderstood that other similar implementations can be used ormodifications and additions can be made to the describedimplementation(s) for performing the same or equivalent function of thecorresponding implementation(s) without deviating therefrom. Stillfurther, multiple processing chips or multiple devices can share theperformance of one or more functions described herein, and similarly,storage can be effected across a plurality of devices. Accordingly, theinvention is not to be limited to any single implementation, but ratheris to be construed in breadth, spirit and scope in accordance with theappended claims.

What is claimed is:
 1. A system, comprising: a processor, and a memorythat stores executable instructions that, when executed by theprocessor, facilitate performance of operations, the operationscomprising: receiving, at a local distributed zone of a distributed zonedata storage system, a client request for requested data from a client;based on determining that the requested data is owned by a first remotedistributed zone, requesting the requested data from the first remotedistributed zone; receiving an indication from the first remotedistributed zone that the requested data is not returnable from thefirst remote distributed zone; and in response to the receiving theindication, obtaining first recovery data, obtaining second recoverydata, combining the first recovery data and the second recovery data toobtain the requested data, and sending, in response to the clientrequest, the requested data from the local distributed zone to theclient.
 2. The system of claim 1, wherein the obtaining the firstrecovery data comprises accessing a storage device of the localdistributed zone, and wherein the obtaining the second recovery datacomprises requesting the second recovery data from a second remotedistributed zone and receiving the second recovery data from a secondremote distributed zone.
 3. The system of claim 2, wherein theoperations further comprise receiving a recovery request from the firstremote distributed zone to recover a copy of a lost or corrupt datastructure owned by the first remote distributed zone that stores therequested data.
 4. The system of claim 1, wherein the obtaining thesecond recovery data comprises requesting the second recovery data froma second remote distributed zone and receiving the second recovery datafrom the second remote distributed zone, and wherein the obtaining thefirst recovery data comprises requesting the first recovery data from athird remote distributed zone and receiving the first recovery data fromthe third remote distributed zone.
 5. The system of claim 1, wherein thecombining the first recovery data and the second recovery data comprisesperforming a bitwise XOR operation of the first recovery data and thesecond recovery data to obtain the requested data.
 6. The system ofclaim 1, wherein the client request for the requested data correspondsto a data segment in a chunk that stores the requested data.
 7. Thesystem of claim 6, wherein the requesting the requested data from thefirst remote distributed zone comprises identifying the chunk, and anoffset value and a size value representing the data segment.
 8. Thesystem of claim 6, wherein the chunk is a first chunk, wherein the firstrecovery data corresponds to a first counterpart data segment in asecond chunk, and wherein the second recovery data corresponds to asecond counterpart data segment in a third chunk in which the thirdchunk comprises a bitwise XOR combination of the first chunk and thesecond chunk.
 9. A method, comprising, receiving, by a system comprisinga processor in a first geographically distributed zone, a request from asecond geographically distributed zone for requested data owned by thefirst geographically distributed zone; determining, by the system in thefirst geographically distributed zone, that the requested data is notreturnable from the first geographically distributed zone; and inresponse to the determining, instructing, by the system in the firstgeographically distributed zone, the second geographic zone to recoverthe requested data.
 10. The method of claim 9, wherein the request is afirst request, wherein the requested data is first requested data, andfurther comprising, receiving, by the system in the first geographicallydistributed zone, a second request from a client requester for secondrequested data owned by the first geographically distributed zone;determining, by the system in the first geographically distributed zone,that the second requested data is not returnable from the firstgeographically distributed zone; and in response to the determining,instructing, by the system in the first geographically distributed zone,the second geographic zone to provide a first recovery part of therequested data, instructing, by the system in the first geographicallydistributed zone, a third geographic zone to provide a second recoverypart of the requested data, receiving, by the system in the firstgeographically distributed zone, the first recovery part, receiving, bythe system in the first geographically distributed zone, the secondrecovery part, recovering, by the system in the first geographicallydistributed zone, the second requested data by combining the firstrecovery part and the second recovery part, and returning, by the systemin the first geographically distributed zone, the second requested datato the client requester in response to the second request.
 11. Themethod of claim 9, wherein the recovering the second requested data bycombining the first recovery part and the second recovery part comprisesperforming an XOR operation.
 12. The method of claim 9, wherein therequested data is part of a corrupt data storage chunk owned by thefirst geographically distributed zone, and further comprising,initiating, by the system in the first geographically distributed zone,recovery of a non-corrupt replacement copy of the corrupt data storagechunk.
 13. The method of claim 9, wherein the requested data is part ofa lost data storage chunk owned by the first geographically distributedzone, and further comprising, initiating, by the system in the firstgeographically distributed zone, recovery of a replacement copy of thelost data storage chunk.
 14. A machine-readable storage medium,comprising executable instructions that, when executed by a processor ofa system in a second distributed zone of a data storage system ofgeographic zones, facilitate performance of operations, the operationscomprising: receiving a client request for requested data owned by afirst distributed zone; in response to the client request, requestingthe requested data from the first distributed zone; receiving anindication from the first distributed zone that the requested data isnot returnable from the first distributed zone; obtaining first recoverydata; obtaining second recovery data from a third distributed zone;combining the first recovery data and the second recovery data to obtainthe requested data; and returning the requested data from the seconddistributed zone in response to the client request.
 15. Themachine-readable storage medium of claim 14, wherein the obtaining thefirst recovery data comprises accessing a storage device of the seconddistributed zone.
 16. The machine-readable storage medium of claim 14,wherein the obtaining the first recovery data comprises requesting andreceiving the first recovery data from a fourth distributed zone. 17.The machine-readable storage medium of claim 14, wherein the combiningthe first recovery data and the second recovery data comprisesperforming a bitwise XOR operation of the first recovery data and thesecond recovery data to obtain the requested data.
 18. Themachine-readable storage medium of claim 14, wherein the receiving theclient request comprises receiving a request for an object thatcorresponds to a data segment in a data chunk owned by the firstdistributed zone.
 19. The machine-readable storage medium of claim 18,wherein the chunk is a first chunk, wherein the obtaining the firstrecovery data comprises accessing a first counterpart data segmentmaintained in a second chunk owned by the second distributed zone, andwherein the obtaining the second recovery data from the thirddistributed zone comprises requesting a second counterpart data segmentmaintained in a third chunk owned by the third distributed zone.
 20. Themachine-readable storage medium of claim 18, wherein the chunk is afirst chunk, wherein the obtaining the first recovery data comprisesrequesting a first counterpart data segment maintained in a second chunkowned by a fourth distributed zone, and wherein the obtaining the secondrecovery data from the third distributed zone comprises requesting asecond counterpart data segment maintained in a third chunk owned by thethird distributed zone.